{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c249b40",
   "metadata": {},
   "source": [
    "# Using Azure OpenAI\n",
    "\n",
    "This tutorial will show you how to use Azure OpenAI endpoints instead of OpenAI endpoints."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e63f667",
   "metadata": {},
   "source": [
    ":::{Note}\n",
    "this guide is for folks who are using the Azure OpenAI endpoints. Check the [evaluation guide](../../getstarted/evaluation.md) if your using OpenAI endpoints.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e54b5e01",
   "metadata": {},
   "source": [
    "### Load sample dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b658e02f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset fiqa (/home/jjmachan/.cache/huggingface/datasets/explodinggradients___fiqa/ragas_eval/1.0.0/3dc7b639f5b4b16509a3299a2ceb78bf5fe98ee6b5fee25e7d5e4d290c88efb8)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5584ed49477f4e788fc4e9b8ab3dc50d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    baseline: Dataset({\n",
       "        features: ['question', 'ground_truths', 'answer', 'contexts'],\n",
       "        num_rows: 30\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data\n",
    "from datasets import load_dataset\n",
    "\n",
    "fiqa_eval = load_dataset(\"explodinggradients/fiqa\", \"ragas_eval\")\n",
    "fiqa_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c77789bb",
   "metadata": {},
   "source": [
    "### Configuring them for Azure OpenAI endpoints\n",
    "\n",
    "Ragas also uses AzureOpenAI for running some metrics so make sure you have your Azure OpenAI key, base URL and other information available in your environment. You can check the [langchain docs](https://python.langchain.com/docs/integrations/llms/azure_openai) or the [Azure docs](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0b7179f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n",
    "os.environ[\"OPENAI_API_VERSION\"] = \"2023-05-15\"\n",
    "os.environ[\"OPENAI_API_BASE\"] = \"...\"\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"your-openai-key\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b8a69c",
   "metadata": {},
   "source": [
    "Lets import metrics that we are going to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f17bcf9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import (\n",
    "    context_precision,\n",
    "    answer_relevancy,\n",
    "    faithfulness,\n",
    "    context_recall,\n",
    ")\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "# list of metrics we're going to use\n",
    "metrics = [\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_recall,\n",
    "    context_precision,\n",
    "    harmfulness,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1201199",
   "metadata": {},
   "source": [
    "Now lets swap out the default `ChatOpenAI` with `AzureChatOpenAI`. Init a new instance of `AzureChatOpenAI` with the `deployment_name` of the model you want to use. You will also have to change the `OpenAIEmbeddings` in the metrics that use them, which in our case is `answer_relevance`.\n",
    "\n",
    "Now in order to use the new `AzureChatOpenAI` llm instance with Ragas metrics, you have to create a new instance of `RagasLLM` using the `ragas.llms.LangchainLLM` wrapper. Its a simple wrapper around langchain that make Langchain LLM/Chat instances compatible with how Ragas metrics will use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40406a26",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.embeddings import AzureOpenAIEmbeddings\n",
    "from ragas.llms import LangchainLLM\n",
    "\n",
    "azure_model = AzureChatOpenAI(\n",
    "    deployment_name=\"your-deployment-name\",\n",
    "    model=\"your-model-name\",\n",
    "    openai_api_base=\"https://your-endpoint.openai.azure.com/\",\n",
    "    openai_api_type=\"azure\",\n",
    ")\n",
    "# wrapper around azure_model\n",
    "ragas_azure_model = LangchainLLM(azure_model)\n",
    "# patch the new RagasLLM instance\n",
    "answer_relevancy.llm = ragas_azure_model\n",
    "\n",
    "# init and change the embeddings\n",
    "# only for answer_relevancy\n",
    "azure_embeddings = AzureOpenAIEmbeddings(\n",
    "    deployment=\"your-embeddings-deployment-name\",\n",
    "    model=\"your-embeddings-model-name\",\n",
    "    openai_api_base=\"https://your-endpoint.openai.azure.com/\",\n",
    "    openai_api_type=\"azure\",\n",
    ")\n",
    "# embeddings can be used as it is\n",
    "answer_relevancy.embeddings = azure_embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44641e41",
   "metadata": {},
   "source": [
    "This replaces the default llm of `answer_relevency` with the Azure OpenAI endpoint. Now with some `__setattr__` magic lets change it for all other metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52d9f5f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "for m in metrics:\n",
    "    m.__setattr__(\"llm\", ragas_azure_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6ecd5a",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Running the evalutation is as simple as calling evaluate on the `Dataset` with the metrics of your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "22eb6f97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluating with [context_recall]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████| 1/1 [00:51<00:00, 51.87s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'context_recall': 0.7750}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ragas import evaluate\n",
    "\n",
    "result = evaluate(\n",
    "    fiqa_eval[\"baseline\"],\n",
    "    metrics=metrics,\n",
    ")\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2dc0ec2",
   "metadata": {},
   "source": [
    "and there you have the it, all the scores you need. `ragas_score` gives you a single metric that you can use while the other onces measure the different parts of your pipeline.\n",
    "\n",
    "now if we want to dig into the results and figure out examples where your pipeline performed worse or really good you can easily convert it into a pandas array and use your standard analytics tools too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8686bf53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truths</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "      <th>context_relevancy</th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>answer_relevancy</th>\n",
       "      <th>context_recall</th>\n",
       "      <th>harmfulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How to deposit a cheque issued to an associate...</td>\n",
       "      <td>[Have the check reissued to the proper payee.J...</td>\n",
       "      <td>\\nThe best way to deposit a cheque issued to a...</td>\n",
       "      <td>[Just have the associate sign the back and the...</td>\n",
       "      <td>0.088301</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.976247</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Can I send a money order from USPS as a business?</td>\n",
       "      <td>[Sure you can.  You can fill in whatever you w...</td>\n",
       "      <td>\\nYes, you can send a money order from USPS as...</td>\n",
       "      <td>[Sure you can.  You can fill in whatever you w...</td>\n",
       "      <td>0.191611</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.883586</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1 EIN doing business under multiple business n...</td>\n",
       "      <td>[You're confusing a lot of things here. Compan...</td>\n",
       "      <td>\\nYes, it is possible to have one EIN doing bu...</td>\n",
       "      <td>[You're confusing a lot of things here. Compan...</td>\n",
       "      <td>0.069420</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.928548</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Applying for and receiving business credit</td>\n",
       "      <td>[\"I'm afraid the great myth of limited liabili...</td>\n",
       "      <td>\\nApplying for and receiving business credit c...</td>\n",
       "      <td>[Set up a meeting with the bank that handles y...</td>\n",
       "      <td>0.408924</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.906223</td>\n",
       "      <td>0.187500</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>401k Transfer After Business Closure</td>\n",
       "      <td>[You should probably consult an attorney. Howe...</td>\n",
       "      <td>\\nIf your employer has closed and you need to ...</td>\n",
       "      <td>[The time horizon for your 401K/IRA is essenti...</td>\n",
       "      <td>0.064802</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.889312</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  How to deposit a cheque issued to an associate...   \n",
       "1  Can I send a money order from USPS as a business?   \n",
       "2  1 EIN doing business under multiple business n...   \n",
       "3         Applying for and receiving business credit   \n",
       "4               401k Transfer After Business Closure   \n",
       "\n",
       "                                       ground_truths  \\\n",
       "0  [Have the check reissued to the proper payee.J...   \n",
       "1  [Sure you can.  You can fill in whatever you w...   \n",
       "2  [You're confusing a lot of things here. Compan...   \n",
       "3  [\"I'm afraid the great myth of limited liabili...   \n",
       "4  [You should probably consult an attorney. Howe...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  \\nThe best way to deposit a cheque issued to a...   \n",
       "1  \\nYes, you can send a money order from USPS as...   \n",
       "2  \\nYes, it is possible to have one EIN doing bu...   \n",
       "3  \\nApplying for and receiving business credit c...   \n",
       "4  \\nIf your employer has closed and you need to ...   \n",
       "\n",
       "                                            contexts  context_relevancy  \\\n",
       "0  [Just have the associate sign the back and the...            0.088301   \n",
       "1  [Sure you can.  You can fill in whatever you w...            0.191611   \n",
       "2  [You're confusing a lot of things here. Compan...            0.069420   \n",
       "3  [Set up a meeting with the bank that handles y...            0.408924   \n",
       "4  [The time horizon for your 401K/IRA is essenti...            0.064802   \n",
       "\n",
       "   faithfulness  answer_relevancy  context_recall  harmfulness  \n",
       "0      0.666667          0.976247        0.111111            0  \n",
       "1      1.000000          0.883586        0.800000            0  \n",
       "2      1.000000          0.928548        1.000000            0  \n",
       "3      1.000000          0.906223        0.187500            0  \n",
       "4      0.666667          0.889312        0.000000            0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result.to_pandas()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f668fce1",
   "metadata": {},
   "source": [
    "And thats it!\n",
    "\n",
    "if you have any suggestion/feedbacks/things your not happy about, please do share it in the [issue section](https://github.com/explodinggradients/ragas/issues). We love hearing from you 😁"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
